Lighting and internet of things design using augmented reality

ABSTRACT

An augmented reality-based lighting design method includes displaying, by an augmented reality device, a real-time image of a target physical area on a display screen. The method further includes displaying, by the augmented reality device, a lighting fixture 3-D model on the display screen in response to a user input, where the lighting fixture 3-D model is overlaid on the real-time image of the target physical area. The method also includes displaying, by the augmented reality device, a lighting pattern on the display screen overlaid on the real-time image of the target physical area, wherein the lighting pattern is generated based on at least photometric data associated with the lighting fixture 3-D model.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority to and is a continuation of U.S.patent application Ser. No. 16/916,526, filed Jun. 30, 2020, which is acontinuation of, and claims priority to, U.S. patent application Ser.No. 15/971,623, filed May 4, 2018 and titled “Lighting and Internet ofThings Design Using Augmented Reality,” which claims priority under 35U.S.C. Section 119(e) to U.S. Provisional Patent Application No.62/608,361, filed Dec. 20, 2017 and titled “Lighting and Internet ofThings Design Using Augmented Reality.” The entire contents of all ofthe preceding applications are incorporated herein by reference. Thisapplication is also related to U.S. patent application Ser. No.15/971,819, filed May 4, 2018 and titled “Lighting and Internet ofThings Design Using Augmented Reality,” the entire content of which isincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates generally to lighting and controlssolutions, and more particularly to lighting or internet of things (IoT)design using augmented reality.

BACKGROUND

A common lighting design method involves examining a target area withrespect to floor plan, ceiling height, structures, etc. and estimatinglighting for the target area using modeling tools. The modeling toolsgenerally rely on 3-D models of the target area that are created basedon the examination of the target area. The generation of the 3-D modelsof the target area and the modeling tools that use the 3-D models can bequite complex. The reliability of the estimated lighting of the targetarea is also heavily dependent on the accuracy of the 3-D models.Similar challenges also exist in IoT design. Thus, a solution thatprovides a user friendly and reliable means of lighting design isdesirable. A similar solution can also be applied in IoT design.

SUMMARY

The present disclosure relates generally to lighting and controlssolutions, and more particularly to lighting or IoT design usingaugmented reality. In an example embodiment, an augmented reality-basedlighting design method includes displaying, by an augmented realitydevice, a real-time image of a target physical area on a display screen.The method further includes displaying, by the augmented reality device,a lighting fixture 3-D model on the display screen in response to a userinput, where the lighting fixture 3-D model is overlaid on the real-timeimage of the target physical area. The method also includes displaying,by the augmented reality device, a lighting pattern on the displayscreen overlaid on the real-time image of the target physical area,wherein the lighting pattern is generated based on at least photometricdata associated with the lighting fixture 3-D model.

In another example embodiment, an augmented reality-based Internet ofThings (IoT) design method includes displaying, by an augmented realitydevice, a real-time image of a target physical area on a display screen.The method further includes displaying, by the augmented reality device,a 3-D model of a lighting fixture with one or more IoT devices on thedisplay screen in response to a user input, where the 3-D model isoverlaid on the real-time image of the target physical area. The methodalso includes displaying on the display screen, by the augmented realitydevice, a pattern overlaid on the real-time image of the target physicalarea, wherein the pattern corresponds to parameter data associated withthe 3-D model.

In another example embodiment, an augmented reality device includes acamera to capture a real-time image of a target physical area and adisplay screen. The device further includes a controller configured toexecute software code to display the real-time image of the targetphysical area on the display screen and to display a lighting fixture3-D model on the display screen in response to a user input, where thelighting fixture 3-D model is overlaid on the real-time image of thetarget physical area. The controller is further configured to executethe software code to display a lighting pattern on the display screen,where the lighting pattern is overlaid on the real-time image of thetarget physical area, wherein the lighting pattern is generated based onat least photometric data associated with the lighting fixture 3-Dmodel.

These and other aspects, objects, features, and embodiments will beapparent from the following description and the appended claims.

BRIEF DESCRIPTION OF THE FIGURES

Reference will now be made to the accompanying drawings, which are notnecessarily drawn to scale, and wherein:

FIGS. 1A and 1B illustrate an augmented reality device for lightingdesign and internet of things (IoT) design according to an exampleembodiment;

FIGS. 1C and 1D illustrate an augmented reality device for lightingdesign and IoT design according to another example embodiment;

FIG. 2 illustrates a block diagram of the augmented reality devices ofFIGS. 1A-1D according to an example embodiment;

FIGS. 3-7A illustrate lighting design stages using the augmented realitydevice of FIGS. 1A-1D according to an example embodiment;

FIG. 7B illustrates luminance levels indicated on the viewport of the ARdevice of FIGS. 1A and 1B according to an example embodiment;

FIG. 7C illustrates a 3-D model of a lighting fixture and lightingpattern including luminance levels based on photometric data or anothergradient of lighting data associated with the lighting fixture accordingto an example embodiment;

FIG. 8A illustrates an e-commerce interface displayed on the augmentedreality device of FIGS. 1A and 1B according to an example embodiment;

FIG. 8B illustrates a bill of material (BOM) generation interfacedisplayed on the augmented reality device 100 of FIGS. 1A and 1Baccording to an example embodiment;

FIG. 8C illustrates a bill of material (BOM) displayed on the augmentedreality device 100 of FIGS. 1A and 1B according to an exampleembodiment;

FIGS. 9-11 illustrate lighting design stages using the AR device ofFIGS. 1A-1D according to another example embodiment;

FIG. 12 illustrates a lighting characteristic selector of the augmentedreality device of FIGS. 1A and 1B according to an example embodiment;

FIGS. 13A-13C illustrate the lighting pattern of FIGS. 11 and 12 withdifferent color temperatures according to an example embodiment;

FIG. 14 illustrates an alternative lighting pattern produced by theaugmented reality device of FIGS. 1A and 1B according to an exampleembodiment;

FIG. 15 illustrates different screenshots of images produced using theaugmented reality device of FIGS. 1A and 1B according to an exampleembodiment;

FIG. 16 illustrates a 3-D model 1602 of a lighting fixture with anintegrated sensor 1606 according to an example embodiment;

FIG. 17 illustrates a 3-D model of a lighting fixture with an integratedcamera 1706 according to an example embodiment;

FIG. 18 illustrates a 3-D model of a lighting fixture with an integratedspeaker 1806 according to an example embodiment;

FIG. 19 illustrates a 3-D model of a lighting fixture with an integratedmicrophone 1906 according to an example embodiment; and

FIGS. 20 and 21 illustrate use of the augmented reality device of FIG.1A to simulate sensor-controlled lighting behavior according to anexample embodiment;

FIG. 22 illustrates a method of augmented reality-based lighting and IoTdesign according to an example embodiment;

FIG. 23 illustrates a method of augmented reality-based lighting and IoTdesign according to another example embodiment;

FIG. 24 illustrates a method of augmented reality-based lighting and IoTdesign according to another example embodiment; and

FIG. 25 illustrates a method of augmented reality-based lighting and IoTdesign according to another example embodiment.

The drawings illustrate only example embodiments and are therefore notto be considered limiting in scope. The elements and features shown inthe drawings are not necessarily to scale, emphasis instead being placedupon clearly illustrating the principles of the example embodiments.Additionally, certain dimensions or placements may be exaggerated tohelp visually convey such principles. In the drawings, the samereference numerals used in different drawings may designate like orcorresponding, but not necessarily identical elements.

DETAILED DESCRIPTION OF THE EXAMPLE EMBODIMENTS

In the following paragraphs, example embodiments will be described infurther detail with reference to the figures. In the description,well-known components, methods, and/or processing techniques are omittedor briefly described. Furthermore, reference to various feature(s) ofthe embodiments is not to suggest that all embodiments must include thereferenced feature(s).

In some example embodiments, an augmented reality (AR) platform may beused by a user, such as lighting designers, consumers, builders,installers, contractors, homeowners, tenants, landlords, buildingoccupants, etc. to place virtual fixture models into a real environmentto quickly gauge appearance as well as view, coordinate, or layoutvarious fixtures lighting parameters such as fixture aesthetic oraccessory options, color temperature, shape, distribution, brightness,light levels, light beam coverage of a space or field of view (e.g., fora camera that may be integrated into the fixture) or sensorrange/direction for sensors (e.g., IR or other type of motion orenvironmental sensor) or accessory devices (speaker range/direction,microphone range/direction) accompanying or separate from a luminaire,etc.

An AR device may include a lighting design AR application and a databaseof lighting fixtures along with associated photometric files orparameter data files with alternative gradient of lighting information.The photometric files (e.g., IES files) contain necessary information toestimate one or more lighting pattern(s) that is produced by lightingfixtures within a three dimensional space. The photometric files mayalso include color temperature, luminance intensity, and/or otherinformation about the light emitted by a lighting fixture. The lightingdesign AR application enables a user to select and place one or morelighting fixtures in a real-time image of a physical/real space beingdisplayed, for example, on a viewport of the AR device and allowsvisualization of how the selected lighting fixture(s) will behave andappear in the physical/real space. The AR application enables a renderedoverlay of the lighting fixture and lighting patterns as well as otherlight characteristics (e.g., color temperature and luminosity) andaccounts for reflected lighting patterns and shadows on surfaces ofobjects/structures in the physical/real space detected by the AR deviceimage processing or other communication between the AR device anddetected objects, which produces reasonably realistic results withoutrequiring installation of actual lighting fixtures. For example, the ARdevice may implement a standalone artificial intelligence application orartificial intelligence code that is integrated with the AR applicationto detect and identify objects/structures in the physical/real space.

Similarly, an AR device may include a sensor (or accessory) design ARapplication and a database of sensors along with associated data files(range, viewing angle, resolution or similar operation information thatmay be visualized through the AR device). For example, the files maycontain necessary information to estimate one or more view angles andrange that is associated with the sensor (e.g., motion, light,temperature, humidity, sound or other type of sensor) or accessorydevice (e.g., camera, microphone, speaker, emitter/detector, wirelessdevice like Bluetooth or WiFi repeater, etc.) within a three dimensionalspace. The files may also include other information about the lightemitted by the sensor or accessory. The AR application enables a user toselect and place one or more sensors or accessories in a real-time imageof a physical/real space being displayed, for example, on a viewport ofthe AR device and allows visualization of how the selected sensors oraccessories will behave and appear in the physical/real space. The ARapplication enables a rendered overlay of the sensors or accessories andassociated patterns or visuals as well as other characteristics. The ARdevice may account for reflected patterns or interference based onsurfaces of objects/structures in the physical/real space detected bythe AR device image processing or other communication between the ARdevice and detected objects, which produces reasonably realistic resultswithout requiring installation of actual sensors or accessories.

FIGS. 1A and 1B illustrate an augmented reality device 100 for lightingdesign according to an example embodiment. In some example embodiments,FIG. 1A illustrates a back side of the augmented reality device 100, andFIG. 1B illustrates the front side of the augmented reality device 100.For example, the augmented reality device 100 may be a tablet, asmartphone, etc. Alternatively, the augmented reality device 100 may bea headset, glasses, goggles, or another type of device with an augmentedreality capable display.

Referring to FIGS. 1A and 1B, in some example embodiments, the augmentedreality (AR) device 100 may include a back-facing camera 102 on a backside of the augmented reality device 100. The AR device 100 may alsoinclude a viewport/display screen 106 on a front side of the augmentedreality device 100. In some example embodiments, the AR device 100 mayalso include a front-facing camera 104, a user input area 108, anambient light sensor 110, accelerometers, or other sensors useful indetermining orientation or real-time feedback from the physical spacethe AR device 100 is located for use in interpreting and displaying theAR on the display 106 of the AR device 100.

In some example embodiments, the viewport 106 may be used to displayimages as seen by the cameras 102, 104 as well as to display objects(e.g., icons, text, etc.) stored, received, and/or generated by the ARdevice 100. The viewport 106 may also be used as a user input interfacefor the AR device 100. For example, the viewport 106 may be a touchsensitive display screen. The viewport 106 may contain a number ofpixels in the vertical and horizontal directions (known as displayresolution). For example, the viewport 106 may have a display resolutionof 2048.times.1536. Each pixel may contain subpixels, where eachsubpixel typically represents red, green, and blue colors.

In some example embodiments, an image of a physical/real area in frontof the AR device 100 may be displayed on the viewport 106 in real timeas viewed by the camera 102. For example, the AR device 100 may includea lighting design AR application that activates the camera 102 such thata real-time image of the physical space viewed by the camera 102 isdisplayed on the viewport 106. Alternatively, the camera 102 may beenabled/activated to display a real-time image of the physical spacebefore or after the lighting design AR application started. In someexample embodiments, the real-time image displayed on the physical spacemay be displayed with a slight delay.

In some example embodiments, the AR device 100 may include an artificialintelligence application and/or component that can determine real lightemitting surfaces and/or other surfaces or structures, such as windows,ceilings, walls, floors, mirrored or reflective surfaces, etc. in aphysical space/area, and automatically suggest/provide recommended typesof lighting fixtures along with additional information such as suggestedlocation, orientation, and/or an appropriate number of lighting fixturesbased on characteristics associated with the light fixtures (e.g.,glare, intensity, available color temperatures or colors, availableoptics or accessories that change the beam angle or distributionproduced by the light fixture, etc.). For example, the artificialintelligence software application and/or component may identify orsuggest the right location for a certain fixture in the observed space,which results in requiring minimal input, interaction, and decisionmaking by a user in achieving lighting design of a physical space/area.Similarly, a software application incorporating suggestions or thatidentifies suggested locations for devices such as sensors (motion,light, environmental conditions like heat, humidity, sound, etc.) oraccessories (e.g., cameras, microphones, speakers, wirelesscommunication, repeaters, etc.) could be used in embodiments aimed atsensors or accessories instead of or in addition to light fixtures.

FIGS. 1C and 1D illustrate augmented reality devices 120, 130 forlighting design and IoT design according to another example embodiment.In some example embodiments, the AR device 120 may be used to performthe operations described above with respect to the AR device 100 in asimilar manner. For example, the glass screens of the devices 120, 130may be used as display screens similar to the viewport 106 of the ARdevice 100. In some example embodiments, another AR device may be usedto perform the operations performed by the AR device 100 in a similarmanner as described above with respect to FIGS. 1A and 1B. Although thedescriptions below are presented generally with respect to the AR device100 of FIGS. 1A and 1B, the description is equally applicable to the ARdevices 120, 130 of FIGS. 1C and 1D.

FIG. 2 illustrates a block diagram of the augmented reality device 100of FIGS. 1A and 1B according to an example embodiment. In some exampleembodiments, the block diagram of FIG. 2 may correspond to the augmentedreality devices 120, 130 of FIGS. 1C and 1D. Referring to FIGS. 1A, 1B,and 2 , in some example embodiments, the AR device 100 includes acontroller 202, a camera component 204, a display component 206, aninput interface 208, a memory device 212, and a communication interface214. For example, the camera component 204 may correspond to or may bepart of the cameras 102, 104. The display component 206 may correspondto or may be part of the viewport/display screen 106 and may includecircuitry that enables or performs displaying of information (e.g.,images, text, etc.) on the viewport 106. For example, the pixels of theviewport may be set/adjusted to display the image as viewed by thecamera 102 or 104. The input interface 208 may correspond to the userinput area 108 and/or the user input capability of viewport 106. Forexample, the display component 206 and the input interface 208 may makeup or may be part of the viewport 106, where the viewport 106 is, forexample, a touch-sensitive display screen. The communication interface214 may be used for communication, wirelessly or via a wired connection,by the AR device 100.

The controller 202 may include one or more microprocessors and/ormicrocontrollers that can execute software code stored in the memorydevice 212. For example, the software code of the lighting design ARapplication and IoT design application may be stored in the memorydevice 212 or retrievable from a remote storage location (e.g., cloudservice or remotely located server or database) via the communicationinterface 214 or other communication means. Other executable softwarecodes used in the operation of the AR device 100 may also be stored inthe memory device 212 or in another memory device of the AR device 100.For example, artificial intelligence lighting and/or other software maybe stored in the memory device 212 as part of the AR application oralong with the AR application and may be executed by the controller 202.

To illustrate, the controller 202 may execute the artificialintelligence application to determine real light emitting surfacesand/or structures (e.g., windows), reflective surfaces, etc. in aphysical space/area, for example, based on the real-time image of thephysical space/area as viewed by the camera 102 or 104 and/or based onlighting condition sensed by an ambient light sensor component 216(corresponding to, connected to, or included in the ambient light sensor110), and automatically suggest/provide recommended type(s) of lightingfixtures along with additional information such as suggested location,orientation, and/or an appropriate number of lighting fixtures. Ingeneral, the one or more microprocessors and/or microcontrollers of thecontroller 202 execute software code stored in the memory device 212 orin another device to implement the operations of the AR device 100described herein. In some example embodiments, the memory device 212 mayinclude a non-volatile memory device and volatile memory device.

In some example embodiments, data that is used or generated in theexecution of the lighting design AR application, IoT design ARapplication, and other code may also be retrieved and/or stored in thememory device 212 or in another memory device of the AR device 100 orretrieved from a remote storage location (e.g., cloud service orremotely located server or database) via the communication interface 214or other communication means. For example, 3-D models of lightingfixtures and photometric data files (e.g., IES files) associated withthe lighting fixture models may be stored in the memory device 112, orretrieved from storage on a remote “cloud”-based service, and may beretrieved during execution of the lighting design AR application. 3-Dmodels of other devices such as sensors, cameras, microphones, speakersemitter/detector, wireless devices such as Bluetooth or WiFi repeater,etc. and data associated with the devices may be stored in the memorydevice 112, or stored in and retrieved from storage on a remote“cloud”-based service, and may be retrieved during execution of IoTdesign AR application on the AR device 100.

The data stored and/or retrieved may include information such as range,viewing angle, resolution or similar operation information that may bevisualized through the AR device). For example, the data may containnecessary information to estimate one or more view angles and range thatis produced by sensor (e.g., motion, light, temperature, humidity, soundor other type of sensor) or an accessory device, such as camera,microphone, speaker, emitter/detector, wireless device like Bluetooth orWiFi repeater, etc. within a three dimensional space. The files may alsoinclude other information about the light emitted by the sensor or theaccessory device.

In some example embodiments, the lighting design AR application storedin the memory device 112 may incorporate or interface with an augmentedreality application/software, such as ARKit, ARCore, HoloLens, etc.,that may also be stored in the memory device 112 or called upon from orprovided via a remote storage location (e.g., cloud service or remotelylocated server or database) via the communication interface 214 or othercommunication means.

The controller 202 may communicate with the different components of theAR device 100, such as the camera component 204, etc., and may executerelevant code, for example, to display a real-time image as viewed bythe camera 102 and/or 104 as well as other image objects on the viewport106.

Although the block diagram of FIG. 2 is described above with respect tothe AR device 100, the block diagram and the above description areequally applicable to the AR devices 120, 130 of FIGS. 1C and 1D.

FIGS. 3-7A illustrate lighting design stages using the augmented realitydevices of FIGS. 1A-1B according to an example embodiment. Although thedescriptions below are presented generally with respect to the AR device100 of FIGS. 1A and 1B, the description is equally applicable to the ARdevices 120, 130 of FIGS. 1C and 1D. In some example embodiments, FIG. 3illustrates a real-time image 304 of a target area 302 displayed on theAR device 100 incorporating the lighting design AR application. Toillustrate, after the lighting design AR application is started, forexample, by selecting a lighting design AR application icon displayed onthe viewport 106, a real-time image 304 of the target area 302 may bedisplayed on the viewport 106. The real-time image 304 displayed on theviewport 106 may be an image of the target area 302 as viewed by theback-facing camera 102. For example, a sofa 306 and a lighting fixture308 that are real objects in the target area 302 are shown in thereal-time image 304. The back-facing camera 102 may be enabled/activatedto view (not necessarily record) the target area 302 in response to theactivation of the lighting design AR application or may beenabled/activated separately.

In some example embodiments, the AR device 100 may be used to assess thetarget area 302 to identify objects, structures, surfaces, etc. in thetarget area 302. For example, the AR device 100 may include and use oneor more accelerometers to determine the orientation of the AR device 100relative to the target area 302, and thus determine orientation ofobjects, structures, surfaces, etc. in the target area 302 based on thereal-time image 304 of the target area 302 as captured by the camera304. The AR device 100 may identify objects, structures, surfaces, etc.by executing artificial intelligence and image processing code and basedon lighting condition of the target area sensed by the ambient lightsensor 110. For example, the AR device 100 may identify light reflective(e.g., mirror), transmissive (e.g., windows), ceilings, walls, floor,furniture, etc. based on the real-time image 304 of the target area 302,the lighting conditions of the target area 302, the orientation of theAR device 100, etc. The AR device 100 may use information from theassessment of the target area 302, for example, to generate displaymodels representing the lighting pattern(s) resulting from selectedlighting fixture models as described below.

In some example embodiments, FIG. 4 illustrates a modified image 402 ofthe target area 302 displayed on the viewport 106 of the AR device 100.For example, a user may provide an input to the AR device 100 (e.g., viathe input area 108 or via the viewport 106) to apply a darkening filterto the pixels of the viewport 106 such that the modified image 402 is aresult of the real-time image 304 and the darkening of the viewport 106.As can be seen in FIG. 4 , the real-time image 304 of the target area302 may still be visible to the user after the darkening filter isapplied. To illustrate, the darkening of the viewport 106 may provide areference lighting level to allow subsequent adjustments of lightingpattern and other characteristics to be more easily discernable.

During the application of the darkening filter to the viewport 106, thepixels of the viewport 106 are transformed based on the pixel data fromthe camera 102 (i.e., the real-time image viewed by the camera 102) andthe light level detected by the ambient light sensor 110. In someexample embodiments, to darken the pixels of the viewport 106, thelighting design AR application may include code corresponding to theequation shown below that is executed by the AR device 100 with respectto the individual pixels of the viewport 106:

PixNew(R,G,B)=PixOld(R,G,B)*DarkFilter(R,G,B), Where:

PixNew(R,G,B) is the pixel resulting from the filtering;PixOld(R,G,B) is the pixel representing the real-time image as viewed bythe camera 102; andDarkFilter=f(Ambient Light Sensor).fwdarw.Z % (e.g., Z=0.1), whereAmbient Light Sensor range is 0 to 255.

By considering the ambient light level, the viewport 106 may be darkenedto a level that allows the real-time image 304 of the target area 302 tobe viewed by user. After the viewport is darkened, the lighting designAR application may display a message to the user indicating the optionof displaying or adding lighting fixtures to the modified image 402.

In some example embodiments, FIG. 5 illustrates the modified image 402displayed on the viewport 106 of the AR device 100 along with lightingfixture 3-D models menu 502 for selection of one or more lightingfixture 3-D models by a user. As described above, each light fixture 3-Dmodel may be stored in a database and may be associated with aphotometric file (e.g., IES file) that includes information indicatinglighting pattern, color temperature, luminance intensity, etc. In someexample embodiments, the lighting fixture 3-D models selectable throughthe menu 502 may include different models of the same type of lightingfixture and/or different types of lighting fixtures, where the differentmodels are associated with respective photometric files representingdifferent lighting patterns, color temperatures, luminance intensity,etc.

In general, the light fixture 3-D models selectable through the menu 502may be provided to the user for selection in one of several other meanssuch as by displaying the models at other locations on the viewport 106,separately on a different display page, as drop-down menu items, etc.Alternatively, the light fixture 3-D models can be selected prior tobringing up the viewport 106 to display the selected light fixture 3-Dmodels in the viewed space.

In accordance with some example embodiments, FIG. 6 illustrates fourlighting fixture 3-D models 602-608 displayed on the viewport 106 alongwith the image 402 shown in FIGS. 4 and 5 . To illustrate, a user mayselect four lighting fixture 3-D models from the lighting fixture modelsmenu 502 provided to the user as shown in FIG. 5 . A user may select thelighting fixture 3-D model 602 from the lighting fixture models menu 502shown in FIG. 5 and place the model 602 at a desired location on themodified image 402. For example, a user may use a finger, stylus, or amouse to select and place the model 602 at a desired location. The usermay select and place the other 3-D lighting fixture 3-D models 604-608at desired locations on the modified image 402 in a similar manner asthe model 602, resulting in the image 610 displayed in the viewport 106shown in FIG. 6 . A user may remove one or more of the models 602-608from the viewport 106, for example, by dragging the particular one moremodels of the viewport 602 or by other means as may be contemplated bythose of ordinary skill in the art with the benefit of this disclosure.

In some example embodiments, when a user places the lighting fixture 3-Dmodels 604-608 at the locations on the modified image 402, the lightingfixture 3-D models 602-608 are associated with physical locations in thetarget area 302 such that the lighting pattern resulting from theselected lighting fixture models 602-608 is shown relative to thephysical locations in the target area 302. For example, the AR device100 may use display coordinates of the viewport 106 to keep track of thephysical locations of the target area corresponding to the locations onthe modified image 402. The AR device 100 may track one or more of tiltangle, orientation, direction, location, distance, etc. of the AR device100 to keep the viewport 106 associated with physical locations of thetarget area 302.

In some example embodiments, FIG. 7A illustrates an image 702 of thetarget area 302 overlaid with a lighting pattern resulting from theselected lighting fixture 3-D models 602-608 and associated photometricfiles. The AR device 100 executes the lighting design AR application toprocess the photometric data of each selected 3-D model 602-608 andgenerate the lighting pattern and the 3-D models 602-608 overlaid on thereal-time image 304 of the target area 302. In some alternativeembodiments, another device, such as a local or remote (e.g., cloud)server, may execute some of the functions of the lighting design ARapplication such as the processing of the photometric data, and providethe resulting information to the AR device 100.

In some example embodiments, the lighting design AR applicationselectively removes/changes the darkening filter applied to the pixels,as necessary, based on the photometric profile (e.g., IES) of theselected lighting fixture 3-D models 602-608. To illustrate, the pixelsof the viewport 106 may be selectively brightened based on thephotometric data corresponding to the selected lighting fixture 3-Dmodels 602-608. For example, pixels of the viewport 106 that are in thelighting distribution area of the selected lighting fixture 3-D models602-608 may be brightened in contrast to the modified image 402 shown inFIG. 4 .

In some example embodiments, the lighting pattern as determined by theAR device 100 may include an area 704 that is well lit as compared toareas 706 and 708 that may be dimly lit. For example, the areas 706, 708may be lit primarily as a result of reflected light from the lightsproduced by the selected lighting fixture 3-D models 602-608. Toillustrate, the lighting design AR application may process thephotometric data of the selected 3-D model 602-608 to determine areasthat may be lit directly and/or as a result of reflected light. Thelighting design AR application may process the photometric data of theselected 3-D model 602-608 to determine the appearance of shadows ondetected or determined surfaces/objects in the real-time image 304 ofthe target area 302, resulting in realistic lighting patterns. Forexample, the AR device 100 may execute an artificial intelligenceapplication to determine objects and structures in the target area, forexample, based on the real-time image of the target area as viewed bythe camera of the AR device 100. For example, the AR device 100 mayidentify reflective surfaces, walls, furniture, etc. and account forreflections, shadows, etc. in removing/changing the darkening filterapplied to the pixels of the viewport 106. In some example embodiments,the AR device 100 also accounts for the lighting conditions in thetarget area, for example, based on lighting conditions sensed by theambient light sensor 110. For example, the AR device 100 may use thelighting condition in the target area to set/adjust parameters used inremoving/changing the darkening filter applied to the pixels of theviewport 106.

In some example embodiments, the AR device 100 may use the photometricdata associated with each selected lighting fixture 3-D model 602-608 togenerate a lighting display model of the lighting pattern that isoverlaid on the real-time image of the target area, resulting in theimage 702 shown in FIG. 7A. The display model of the lighting pattern(including luminance levels, color temperature, etc.) may be a polygonor another type of image. In some alternative embodiments, the AR device100 may send information indicating the selected lighting fixture 3-Dmodels 602-608 to another processing device, such as a local or cloudserver, and the other processing device may generate a display model(e.g., a polygon or another image) based on the photometric dataassociated with the respective lighting fixture 3-D models 602-608. TheAR device 100 may receive or retrieve the generated display model fromthe other processing device for display on the viewport 106, where thedisplay model is overlaid on the real-time image of the target area 302.

In some example embodiments, the display model may be a polygon, such asa 2-dimensional (2D) polygon, a 3-dimensional (3-D) polygon, acombination of 2D and/or 3-D polygons, etc., or one or more other typesof images such as graphical images, etc. To illustrate, the imagedisplayed in FIG. 7A may be a result of the AR device 100 overlaying alighting display model over the image 610 shown in FIG. 6 , whicheffectively removes/changes the darkening filter shown in FIG. 5 . Forexample, the AR device 100 may generate or retrieve a polygon that hasdisplay parameters corresponding to the lighting pattern represented bythe photometric data files associated with the multiple selectedlighting fixture 3-D models 602-608. Information such as colortemperature, luminance levels, etc. contained in the photometric datafiles may be represented by the display parameters of the polygon, andthe pixels of the viewport 106 may be changed/set based on theseparameters. Different points or parts of the generated polygon may beassociated with different luminance levels, color temperature values,etc. contained in the photometric data files. The AR device 100 maydisplay the real-time image of the target area overlaid with the polygonby adjusting/setting the pixels of the viewport 106. For example, thedisplay of the generated polygon on the viewport 106 may remove/changethe darkening filter applied at the design stage shown in FIG. 5 ,resulting in the image shown in FIG. 7A.

In some example embodiments, the AR device 100 may generate or retrievea display model, such as a polygon (e.g., a 2D polygon, a 3-D polygon, acombination of 2D and/or 3-D polygons, graphical images, etc.) oranother type of image(s), for each one of the selected lighting fixture3-D model 602-608 and combine the multiple display models to generate adisplay model representing the combined lighting pattern. For example,the AR device 100 may combine polygons that have parameterscorresponding to the photometric data of each selected lighting fixture3-D model 602-608 to generate a combined polygon that has displayparameters that account for the display parameters of the individualpolygons. The AR device 100 may retrieve the individual polygons orother types of display models from a local storage or a remote sourcesuch as a cloud server.

In some example embodiments, the AR device 100 may account for lightingconditions in the target area in generating the display modelrepresenting the lighting pattern resulting from the selected lightingfixture 3-D model 602-608. For example, the AR device 100 may use thelighting condition sensed by the ambient light sensor 110 as well as thephotometric data of each selected lighting fixture 3-D model 602-608 togenerate the display parameters of a polygon that is displayed on theviewport 106 overlaid on the real-time image of the target area 302. TheAR device 100 may identify reflective surfaces, walls, furniture, etc.as described above and account for reflections, shadows, etc. ingenerating the polygon that is overlaid on the real-time image.

As illustrated in FIG. 7A, the selected lighting fixture 3-D models602-608 are displayed in the real-time image of the target area 302,enabling the user to assess how the corresponding lighting fixtures orlighting effect will look when installed in the target area 302. Usingthe AR device 100, a user (e.g., a lighting designer, owner, etc.) maymore effectively perform lighting design of a particular area (e.g., aliving room, a bedroom, a hallway, office, warehouse, an outdoorlandscape, a parking lot, etc.) without having to install actuallighting fixtures and at the same time minimizing design errors. Becausethe selected lighting fixture models 602-608 are associated with thephysical locations of the target area 302 as described above and becausethe lighting display models (e.g., the polygon(s)) are associated withthe selected lighting fixture models 602-608, a user may move in thetarget area 302 holding the AR device 100 and assess the placements ofthe lighting fixtures and the resulting lighting effect at differentlocations in the target area 302. As the user moves through and near thetarget area 302, the shape of the lighting pattern displayed on theviewport 106 may change depending on the part of the target areaviewable by the camera 102 of the AR device 100 and the correspondingreal-time image displayed on the viewport 106.

As described above, a display model that represents the photometric dataassociated with one or more lighting fixtures may be a 2D polygon, a 3-Dpolygon, and a combination of 2D and/or 3-D polygons, graphicalimage(s), another type of image(s), etc. In general, a polygon that isused as a display model may be a 2D polygon, a 3-D polygon, acombination of 2D and/or 3-D polygons, graphical image(s), another typeof image(s), etc.

In some example embodiments, a user may change the outward appearances(e.g., color) of the lighting fixture 3-D models 602-608 withoutchanging lighting characteristics (e.g., luminance level, colortemperature, etc.) associated with the lighting fixture 3-D models602-608. For example, in response to a user input (e.g., clicking ortapping on a displayed lighting fixture 3-D model), the AR device 100may change the color of the trim ring and/or the color of the housing ofthe displayed lighting fixture 3-D model without changing the lightingpattern displayed on the viewport 106. For example, clicking or tappingon a displayed lighting fixture 3-D model by a user may result in the ARdevice 100 executing software code to change the color of the housing ina predefined order (e.g., white, blue, red, white, . . . ).

In some example embodiments, a user may use the AR device 100 to assessthe appearance of the corresponding lighting fixtures in the target area302. For example, the AR device 100 may overlay the lighting fixture 3-Dmodels 602-608 in the real-time image 304 of the target area 302 toassess the appearance of the corresponding lighting fixtures in thetarget area 302 without installing the lighting fixtures. To illustrate,after the real-time image 304 is displayed on the viewport 106 as shownin FIG. 3 , the AR device 100 may overlay the lighting fixture 3-Dmodels 602-608 on the real-time image 304 in response to a user input.For example, the lighting fixture 3-D models menu 502 may be displayedon the viewport 106 along with the image 304. A user may select andplace the lighting fixture 3-D models 602-608 and/or other 3-D models onthe real-time image 304. In some example embodiments, the lightingpatterns associated with the lighting fixture 3-D models 602-608 andother 3-D models may or may not be displayed on the viewport 106 whenthe AR device 100 is used to assess the physical appearance of lightingfixtures. For example, the design stages associated with FIG. 4 andsubsequent generation and/or display of a lighting pattern may beomitted.

As described above, the color of a trim ring, size of the trim ring,type of trim ring or alternative optical attachment, lens type, thecolor of a lighting fixture housing, alternative subcomponent(s) of thelight fixture, and/or other aesthetic aspects of a displayed lightingfixture 3-D model may be changed, for example, by tapping or clicking onthe displayed lighting fixture 3-D model. In some alternativeembodiments, aesthetic features of displayed lighting fixture 3-Dmodels, such as the 3-D models 602-608, may be changed after thelighting patterns associated with the lighting fixture 3-D models aredisplayed, for example, as shown in FIG. 7A.

In general, the lighting design AR application executed by the AR device100 may include or rely on operations performed by AR applications, suchas ARKit, ARCore, etc. In some alternative embodiments, a still image (acaptured picture) of the target area 302 may be used instead of areal-time image. For example, a photograph that contains adequateinformation, such as tilt angle of the AR device 100, GPS location, etc.may allow the AR device 100 executing the lighting design AR applicationand/or an artificial intelligence application to determine 3-Dinformation from the photograph and enable lighting design based on theinformation.

In some alternative embodiments, another device may perform some of theoperations described herein with respect to the AR device 100. Toillustrate, another device, such as a local or remote server, maygenerate one or more display models based on information provided by theAR device 100. For example, the AR device 100 may provide informationsuch as the selected lighting fixture 3-D model 602-608 and/or relevantphotometric data to another processing device that generates the displaymodel(s), and the AR device 100 may receive/retrieve the generateddisplay model(s) from the other processing device.

FIG. 7B illustrates luminance levels indicated on the viewport 106 ofthe AR device 100 of FIGS. 1A and 1B according to an example embodiment.In some example embodiments, luminance level values may be displayed onthe viewport 106, for example, to provide a numeric representation ofbrightness levels at different locations of the target area based on theselected lighting fixture 3-D model 602-608. For example, differentpoints or areas of a display model (e.g., different points or areas of apolygon) generated as described above may be associated or otherwisetagged with the luminance level values. To illustrate, some areas may beassociated with higher brightness level (e.g., 5.5 foot-candle (FC))while other areas may be associated with a relatively darker level(e.g., 3.2 FC). As a user moves in the target area holding the AR device100, the luminance level values that are displayed may change dependingon the part of the target area that is viewed by the camera of the ARdevice 100 and displayed on the viewport 106 based on the location ofthe user relative to the selected lighting fixture 3-D model 602-608.

FIG. 7C illustrates a 3-D model of a lighting fixture 700 and lightingpattern including luminance levels that are based on photometric data oranother gradient of lighting data associated with the lighting fixtureaccording to an example embodiment. For example, the lighting fixture700 may correspond to the lighting fixtures 602-608 shown in FIG. 7A.The photometric data associated with the lighting fixture 700 may beillustrated to convey lighting distribution shape, color temperature aswell as the luminance levels indicated by the luminance level values,for example, at a surface that is a particular distance from thelighting fixture 700. Although the luminance level values are shown fora particular surface, the photometric data may include luminance levelvalues at different distances. The AR device 100 may use the photometricdata including lighting distribution shape, color temperature, theluminance levels, etc. to generate a display model that is overlaid onthe real-time image of the target area displayed on the viewport 106.Although a polygon is described herein as an example of a display model,other types of display models such as other types of images may equallybe used.

FIG. 8A illustrates an e-commerce interface 802 displayed on theaugmented reality device 100 of FIGS. 1A and 1B according to an exampleembodiment. Referring to FIGS. 1A-8A, the information included in thee-commerce interface 802 may be generated based on the lighting designstages described above. To illustrate, in some example embodiments, theuser may be given an option to purchase the lighting fixturescorresponding to the selected lighting fixture 3-D models 602-608. Forexample, the AR device 100 may execute the AR application to display aweblink on the viewport 106 for a user to click or tap to purchase thelighting fixtures corresponding to the selected lighting fixture 3-Dmodels 602-608. Alternatively, the weblink may be provided to the useron a separate web browser page or in a separate e-commerce applicationscreen when the design stages are completed and/or the display of ARrelated information is terminated by the user. Other purchasing optionsincluding the option to make the purchase via voice command, etc. mayalso be provided to the user. For example, the lighting design ARapplication may incorporate or interface with another application toprovide the purchasing option as well as to execute the purchase of thelighting fixtures based on the user's input.

In some alternative embodiments, the e-commerce interface 802 may bedisplayed in a different format than shown in FIG. 8A without departingfrom the scope of this disclosure. In some alternative embodiments,other user input icons and information may be displayed on the viewportwithout departing from the scope of this disclosure. Although thee-commerce interface 802 is described above with respect to the ARdevice 100 of FIGS. 1A and 1B, the description is equally applicable tothe AR devices 120, 130 of FIGS. 1C and 1D.

FIG. 8B illustrates a bill of material (BOM) generation input interface806 displayed on the augmented reality device 100 of FIGS. 1A and 1Baccording to an example embodiment. In some example embodiments, a usermay use the BOM generation input interface 806 to generate a BOM (orpurchase order for the BOM) or, more generally, a list of productsavailable for purchase (including, in some embodiments, any accessoriesor additional items required for installation or operation) resultingfrom the AR-based design described above. For example, following thedisplay of the image 702 shown in FIG. 7A, the BOM generation page maybe displayed in the viewport 106 as shown in FIG. 8B. A user may tap orclick on the BOM generation input interface 806, and, in response, theAR device 100 may execute the AR application or another application togenerate a BOM that includes, for example, identification information(e.g., model number, product number, etc.) that corresponds to thelighting fixture 3-D models 602-608 shown in FIG. 7A and/or otherlighting fixtures and devices added by the user (including, in someembodiments, any accessories or additional items required forinstallation or operation). The BOM generation page shown in FIG. 8B maybe presented on the viewport 106 prior to the display of the e-commerceinterface 802 shown in FIG. 8A. For example, the e-commerce interface802 shown in FIG. 8A may be displayed on the viewport 106 following thegeneration of a BOM.

In some example embodiments, a product menu 804 may also be displayed onthe viewport 106. For example, the product menu 804 may allow a user toadd additional products to a BOM. The product menu 804 may allow a userto add lighting fixtures with or without integrated IoT devices (e.g.,sensors, camera, speakers, microphones, etc.), load control devices(e.g., relays, switches, dimmers, etc.), IoT devices (e.g., standaloneconnected sensors, microphones, a speaker, etc.), trims, junction boxes,wall-stations, and other types of products and any accessories oradditional items required for installation or operation (e.g., wireharness, connectors, cables, remote power supplies, etc.) to thegenerated BOM. As used herein IoT device refers to any sensor and/orcommunication device that may be integrated into a light fixture or maybe a standalone device that is capable of controlling or otherwisecommunicating with or to a light fixture or other device located in thevicinity of the IoT device or providing communications for a lightfixture or other device in the vicinity of the IoT device to a network.Alternatively or in addition, the product menu 804 may allow a user toadd additional products prior to the generation of a BOM. To illustrate,following the design stages corresponding to FIG. 7A or FIG. 7B, a usermay add other products (e.g., a load control device, etc.) using theproduct menu 804 prior to the generation of a BOM.

In some example embodiments, the product menu 804 may be a drop downmenu, another type of user interface (e.g., a list), a link to anotherpage, etc. In some example embodiments, a product search interface mayalso be presented instead of or in addition to the product menu 804. Insome alternative embodiments, the BOM generation input interface 806 maybe displayed on the viewport 106 at different design stages such as atthe design stages corresponding to FIGS. 6-7C. In some alternativeembodiments, the BOM generation input interface 806 may be displayed ata different location of the viewport 106 and/or or may be displayed orprovided to the user in a different format such as a selection from adrop-down menu, etc.

FIG. 8C illustrates a bill of material (BOM) 808 displayed on theaugmented reality device 100 of FIGS. 1A and 1B according to an exampleembodiment. Referring to FIGS. 1-8B, in some example embodiments, theBOM 808 may be generated by the AR device 100 in response to the userinput provided via the BOM generation input interface 806. For example,a user may use the BOM generation input interface 806 displayed on theviewport 106 as shown in FIGS. 8B, 8C, or as can be displayed at otherdesign stages such as the design stages corresponding to FIGS. 7A and7B.

In some example embodiments, after the BOM 808 is generated anddisplayed, a user may add additional products such as lighting fixtureswith or without integrated IoT devices, load control devices, IoTdevices, trims, junction boxes, wall-stations, and other types ofproducts to the generated BOM 808. For example, a user may use theproduct menu 804 to add additional products to the generated BOM 808 asdescribed above with respect to FIG. 8B.

In some example embodiments, a user may request validation of the BOM808 by providing an input using the BOM validation input interface 812.For example, clicking or tapping the BOM validation input interface 812may result in the BOM 808 being sent to a technical support person, acontractor, a sales representative, or automated validation system incommunication with the AR device that can confirm the accuracy,completeness, or availability of the items listed on the BOM. Thetransmission of the BOM 808 by the AR device 100 may be performed byexecuting the AR application and/or another software code or applicationas can be readily understood by those of ordinary skill in the art withthe benefit of this disclosure. Alternatively or in addition to sendingthe BOM 808, clicking or tapping the BOM validation input interface 812may initiate a chat session with a technical support person, acontractor, a sales representative, etc.

In some example embodiments, clicking or tapping the BOM validationinput interface 812 may initiate operations by the AR device 100 toverify design information 814, which may include whether the productsincluded in the BOM 808 are compliant with one or more lighting orelectrical codes and/or guidelines. For example, the lighting orelectrical codes and/or guidelines may be international, national,and/or local codes and guidelines. To illustrate, the lighting orelectrical codes and/or guidelines may address light levels relevant toparticular spaces (e.g., OSHA guidelines, etc.), lighting fixturestandby power and startup time (e.g., Title 24 of the California Code ofRegulations, etc.), plenum rating (e.g., City of Chicago ElectricalCode, etc.), and other electrical and lighting requirements andguidelines such as those included in European Union standards.

In some example embodiments, one or more lighting and/or electricalcodes and/or guidelines may be stored in the memory device 212 oranother memory device. Alternatively or in addition, one or morelighting and/or electrical codes and/or guidelines may be retrieved orcompared for compliance by the AR device 100 from a remote source inresponse to a user input provided to the AR device 100 via the BOMvalidation input interface 812 or another user interface. For example,the AR device 100 may retrieve relevant lighting and/or electrical codeand/or guidelines or compare compliance with such guidelines based ongeographic location information provided by a user or based on alocation of the AR device 100 determined by the AR device 100 using GPSand/or other means.

In some example embodiments, the AR device 100 may display other designinformation 814 on the viewport 106. For example, the design information814 may include information indicating whether the products in the BOM808 are compliant with one or more codes and/or guidelines such as thosedescribed above. The AR device 100 may display design information 814 inresponse to the user input provided using the BOM validation inputinterface 812. Alternatively or in addition, the AR device 100 maydisplay design information 814 in response to the generation of the BOM808 as described above. In some example embodiments, the AR device 100or via communication with a cloud sever having access to inventoryinformation, may display whether or not one or more products in the BOM(e.g., the BOM 808) are available for purchase or an estimate of whenthe one or more products may be available for purchase or delivery.

In some example embodiments, the design information 814 may includesuggestions of additional and/or replacement products. For example, thedesign information 814 may suggest one or more load control devices(e.g., relays, etc.) based on the number lighting fixtures and IoTdevices included in the BOM 808 and the power ratings of the lightingfixtures and IoT devices. As another example, the design information 814may suggest one or more replacement lighting fixtures to meet lightlevel guidelines and/or requirements, occupancy-based lighting controlrequirements, plenum rating requirements, power density requirements,etc. In some example embodiments, the design information 814 may provideinformation indicating wire gauge recommendations based the number oflighting fixtures and load control devices included in the BOM 808. Auser may use the product menu 804 to add products to the BOM 808 or toreplace products included in the BOM 808.

In some example embodiments, the user may order the products included inthe BOM 808 using the order input interface 810. For example, clickingor tapping the order input interface 810 may result in the e-commerceinterface 802 or another page/interface being displayed on the viewport106 for the execution of a purchase/ordering of the products included inthe BOM.

In general, the AR device may execute software code included in the ARapplication or interfaced with the AR application to perform theoperations described herein. Alternatively or in addition, the AR device100 may send relevant information to another device (e.g., a cloudserver) to perform some of the operations.

In some alternative embodiments, the BOM 808, interfaces, etc. shown inFIG. 8C may be displayed in a different format, on different pages, etc.without departing from the scope of this disclosure. In some alternativeembodiments, one or more of the interfaces and information shown in FIG.8C may be omitted without departing from the scope of this disclosure.

FIGS. 9-11 illustrate lighting design stages using the AR device 100 ofFIGS. 1A-1D according to another example embodiment. Although thedescriptions below are presented generally with respect to the AR device100 of FIGS. 1A and 1B, the description is equally applicable to the ARdevices 120, 130 of FIGS. 1C and 1D. In some example embodiments, FIG. 9illustrates a real-time image 902 of a target area displayed on theviewport 106 of the AR device 100 incorporating the lighting design ARapplication. FIG. 10 illustrates a modified image 1002 of a target areaalong with lighting fixture 3-D models 1004 displayed for selection by auser. For example, as described with respect to FIG. 4 , a user mayprovide an input to the AR device 100 (e.g., via the input area 108 orvia the viewport 106) to apply a darkening filter to the pixels of theviewport 106 such that the modified image 1002 is a result of thereal-time image 902 and the darkening of the viewport 106. As can beseen in FIG. 10 , the real-time image 902 of the target area may stillbe visible to the user after the darkening filter is applied to allowthe user to place one or more selected lighting fixture 3-D models at adesired location with respect to the real-time image 902 of the targetarea.

In some example embodiments, FIG. 11 illustrates an image 1104 thatincludes a real-time image 902 of the target area overlaid with alighting pattern associated with or resulting from a selected lightingfixture 3-D model 1102 and an associated photometric file. The AR device100 executes the lighting design AR application to process thephotometric data of the selected 3-D model 1102 and generate thelighting pattern, which is overlaid on the real-time image 902 of thetarget area along with the selected lighting fixture 3-D model 1102. Forexample, the AR device 100 may use the photometric data associated withthe selected lighting fixture 3-D model 1102 to generate a display model(e.g., a polygon or another display model) representing the lightingpattern, and the generated display model may be displayed on theviewport 106 overlaid on the real-time image 902 of the target area. Insome alternative embodiments, the display model may bereceived/retrieved by the AR device 100 in response to the selection ofthe lighting fixture 3-D model 1102. For example, the AR device 100 mayreceive/retrieve the display model representing the lighting pattern(e.g., as a polygon or another type of display model) from anotherdevice (e.g., a local or remote server) that generates the display modelafter receiving from the AR device 100 information indicating theselected lighting fixture 3-D model 1102.

Information such as color temperature, luminance levels, etc. containedin the photometric data may be represented by the parameters of thedisplay model, and the pixels of the viewport 106 are changed/set basedon the parameters of the display model. For example, different points orparts of a polygon (or another display model) may be associated withdifferent luminance levels, color temperature values, etc. contained inthe photometric data associated with the selected lighting fixture 3-Dmodel 1102. The AR device 100 may display the real-time image of thetarget area overlaid with the polygon by adjusting/setting the pixels ofthe viewport 106 to account for the parameters of the polygon.

In some example embodiments, the AR device 100 may use the photometricdata associated with the selected lighting fixture 3-D model 1102 alongwith the lighting conditions in the target area to generate a polygon(or another display model) that has parameters that are based on boththe photometric data and the lighting conditions. For example, the ARdevice 100 may use the lighting condition sensed by the ambient lightsensor 110 to generate the parameters of a display model. In someexample embodiments, the AR device 100 may generate a display modelbased on the photometric data of the selected lighting fixture 3-D model1102 and modify the parameters of the display model based on the sensedlighting condition.

In some example embodiments, the AR device 100 may execute an artificialintelligence application to determine objects and structures in thetarget area, for example, based on the real-time imager of the targetarea. For example, the AR device 100 may identify reflective surfaces,walls, furniture, etc. and account for reflections, shadows, etc. ingenerating the display model that is overlaid on the real-time imagedisplayed on the viewport 106.

The AR device 100 executes the lighting design AR application toselectively remove/change the darkening filter applied to the pixels ofthe viewport 106 as described above with respect to FIG. 7 . A shown inFIG. 11 an area 1106 may be well lit as compared to areas 1108 and 1110that may be dimly lit. The lighting design AR application may processthe photometric data of the selected 3-D model 1102 to determine theappearance of shadows, etc., resulting in realistic lighting patterns.

As illustrated in FIG. 11 , the selected lighting fixture 3-D model 1102is displayed in the real-time image of the target area, enabling theuser to assess how the corresponding lighting fixture will look wheninstalled in the target area. Thus, a user, such as a lighting designer,owner, etc., may more effectively perform lighting design of aparticular area (e.g., a living room, a bedroom, a hallway, etc.)without having to install actual lighting fixtures while minimizingdesign errors.

As described above, a display model that represents the photometric dataassociated with one or more lighting fixtures may be a 2D polygon, a 3-Dpolygon, a combination of 2D and/or 3-D polygons, graphical image(s),another type of image(s), etc. A polygon as an example of a displaymodel may be a 2D polygon, a 3-D polygon, a combination of 2D and/or 3-Dpolygons, graphical image(s), another type of image(s), etc.

In some alternative embodiments, another device may perform some of theoperations described herein with respect to the AR device 100. Toillustrate, another device, such as a local or remote server, maygenerate one or more display models based on information provided by theAR device 100. For example, the AR device 100 may provide informationsuch as the selected lighting fixture 3-D model 602-608 and/or relevantphotometric data to another processing device that generates the displaymodel(s), and the AR device 100 may receive/retrieve the generateddisplay model(s) from the other processing device.

FIG. 12 illustrates a lighting characteristic selector 1202 of theaugmented reality device of FIG. 1 according to an example embodiment.In some example embodiments, the lighting characteristic selector 1202may be used to change and/or select a color temperature or even aparticular color (e.g., wavelength) of the light from the 3-D model1102. For example, the selector 1202 may be used to select from discretecolor temperature values corresponding to lighting fixture 3-D modelsstored or available to the AR device 100. Alternatively or in addition,the selector 1202 or another selector may be used to change and/orselect luminance levels, light output patterns of the selected 3-D model1102, etc. In some example embodiments, changing a characteristic of thelight from the selected 3-D model 1102 may effectively result in areplacement of the selected 3-D model by another 3-D model or in thereplacement of the associated photometric file(s), for example, when theoutward appearance of the 3-D model is not affected by the changes. Thechange/selection of a characteristic of the light from the selected 3-Dmodel 1102 may result in a corresponding change being reflected in theimage 1104 displayed on the viewport 1104.

FIGS. 13A-13C illustrate the lighting pattern of FIGS. 11 and 12 withdifferent color temperatures according to an example embodiment. Thelighting pattern with the particular color temperature shown in FIGS.13A-13C may be produced by the AR device based on a user selection of arespective color temperature using, for example, the lightingcharacteristics selector 1202 of FIG. 12 or another selection means. Forexample, the CCT of 2200 K shown in FIG. 13A may correspond to thebottom most position of the lighting characteristics selector 1202, theCCT of 4000 K shown in FIG. 13B may correspond to a middle position ofthe lighting characteristics selector 1202, and the CCT of 6500 K shownin FIG. 13C may correspond to the upper most position of the lightingcharacteristics selector 1202. To illustrate, in response to a CCTselection indicated by the lighting characteristics selector 1202, theAR devices 100 may execute code to change the color (and as needed, thebrightness) of relevant pixels. For example, subpixels of each relevantpixel may be adjusted to produce a desired color of the particularpixel. In some example embodiments, the AR device 100 applies adarkening, brightening, and/or color adjustment filter to the pixels ofthe viewport 106 to display the lighting pattern with the CCTcorresponding to the selection indicated by the lighting characteristicsselector 1202.

In some example embodiments, particular positions of the lightingcharacteristics selector 1202 may be associated with a respectivedisplay model stored in or otherwise retrievable by the AR device 100.For example, each model may be a polygon that has a shape correspondingto a particular light distribution pattern, where the polygon hasdisplay parameters corresponding to a CCT value, etc. To illustrate, theAR device 100 may modify the pixels of the viewport 106 to display thepolygon (i.e., the display model) overlaid on the real-time image of thetarget area. In some example embodiments, the AR device 100 may generateor retrieve the CCT related parameters of the polygon based on the CCTindicated by the lighting characteristics selector 1202. In some exampleembodiments, the AR device 100 may generate or modify the parameters ofthe polygon based on the CCT selection indicated by the lightingcharacteristics selector 1202 along with the lighting condition in thetarget area, for example, sensed by the ambient light sensor 110 of thedevice 100.

In some alternative embodiments, each color temperature of the lightingpattern shown in FIGS. 13A-13C may be produced by selecting a lightingfixture 3-D model from among the 3-D models 1004 shown in FIG. 10 ,where the selected 3-D model is associated with a photometric file thatreflects the desired color temperature.

In some example embodiments, the lighting pattern of FIGS. 11 and 12 mayalso be produced with a desired luminance level in a similar manner asdescribed with color temperature. To illustrate, the lighting pattern ofFIGS. 11 and 12 may also be produced with a desired luminance level byselecting the desired luminance level using the selector 1202 of FIG. 12or another means. For example, a top position of the lightingcharacteristics selector 1202 may correspond to a first luminance level(e.g., 1300 lumens), a middle position of the lighting characteristicsselector 1202 may correspond to a second luminance level (e.g., 1100lumens), and a bottom position of the lighting characteristics selector1202 may correspond to a third luminance level (e.g., 900 lumens). Toillustrate, in response to a luminance level selection indicated by thelighting characteristics selector 1202, the AR devices 100 may executecode to change the color (and as needed, the brightness) of relevantpixels. In some example embodiments, the AR device 100 applies adarkening or brightening filter to the pixels of the viewport 106 todisplay the lighting pattern with the luminance levels corresponding tothe selection indicated by the lighting characteristics selector 1202.

In some example embodiments, particular positions of the lightingcharacteristics selector 1202 may be associated with a respectivedisplay model stored in or otherwise retrievable by the AR device 100.For example, each display model may be a polygon that has a shapecorresponding to a particular light distribution pattern, where thepolygon has display parameters corresponding to luminance levels, etc.To illustrate, the AR device 100 may modify the pixels of the viewport106 to display the polygon (i.e., the display model) overlaid on thereal-time image of the target area. In some example embodiments, the ARdevice 100 may generate or retrieve the luminance level relatedparameters of the polygon based on the luminance level indicated by thelighting characteristics selector 1202. In some example embodiments, theAR device 100 may generate or modify the parameters of the polygon basedon the luminance level selection indicated by the lightingcharacteristics selector 1202 along with the lighting condition in thetarget area, for example, sensed by the ambient light sensor 110 of thedevice 100.

Alternatively, the desired luminance intensity may be achieved byselecting a 3-D model associated with a photometric file that includesthe desired luminance intensity.

FIG. 14 illustrates an alternative lighting pattern produced by the ARdevice 100 according to an example embodiment. The alternative lightingpattern may result from the selection of a lighting fixture 3-D model1402 that produces a different lighting pattern as compared to the 3-Dmodel 1102 of FIG. 11 .

In some example embodiments, the color temperature, luminance intensity,lighting pattern, and/or another characteristic of the light from alighting fixture 3-D model may be changed after the initial lightingpattern as shown in FIG. 11 (also in FIG. 7A) is displayed.

FIG. 15 illustrates an image frame 1500 screenshots of images producedusing the augmented reality device of FIGS. 1A and 1B according to anexample embodiment. These images may also be displayed on a screen ofthe AR devices 120, 130 of FIGS. 1C and 1D. In some example embodiments,going from left to right of each row of images, in the top row, theleftmost image shows an icon (LiAR) for activating the AR applicationdescribed above. The next image in the top row shows a menu that can beused, for example, to select a lighting fixture model to be placed in areal-time image of the area viewable by the camera of the AR device. Theright two images on the top row and all the images in the middle rowshow 3-D models of different lighting fixtures overlaid on the image ofthe physical area (e.g., an office hallway) as viewed by the camera ofthe AR device.

The images on the bottom row show 3-D models of different outdoorlighting fixtures overlaid on the image of the physical area (e.g., awalkway) as viewed by the camera of the AR device. In general, the ARdevices 100, 120, 130 may execute AR lighting design application tooverlay one or more 3-D models of indoor and outdoor lighting fixtureson images of physical spaces (e.g., indoor space such as living room,kitchen, hallway, halls, etc. and outdoor spaces such as parkinggarages, open parking lots, walkways, stadiums, auditoriums, etc. tomake a realistic assessment of the appearance of the lighting fixturesas well as the lighting effects of the lighting fixtures prior to theinstallation of the lighting fixtures.

FIG. 16 illustrates a 3-D model 1602 of a lighting fixture with anintegrated sensor 1606 according to an example embodiment. In someexample embodiments, the AR devices 100, 120, 130 may be used in IoTdesign (e.g., placement of IoT devices in a space) in addition to orinstead of the lighting design. The AR devices 100, 120, 130 may be usedto perform IoT design in a similar manner as described above withrespect to the AR device 100 and lighting design. For example, a menu1604 of 3-D models of lighting fixtures with and without integrated IoTdevices (e.g., the sensor 1606) and standalone IoT devices may bedisplayed or otherwise provided on the viewport 106 of the AR device 100and on the corresponding display screen of the AR devices 120 and 130.Each 3-D model may be associated with a parameter data file thatincludes data indicating the detection range/angles/field of view 1608of the sensor 1606 (e.g., a motion sensor, carbon dioxide sensor, carbonmonoxide sensor, smoke sensor, etc.). A user may select a 3-D model suchas the 3-D model 1602 of a lighting fixture with the sensor 1606 andplace the 3-D model on a desired location on the real-time image of atarget area as described above with respect to FIGS. 3-11 . The ARdevices 100, 120, 130 may execute the AR application to overlay on thereal-time image of the target area a display model (e.g., a polygon oranother display model) corresponding to the detection range/angle of thesensor 1606 in a similar manner as described above with respect tolighting design. For example, the AR device 100, 120, 130 may generatethe display model or retrieve an existing display model associated withthe selected 3-D model 1602. In some example embodiments, the parameterdata file may include other information that can be used to generate thedisplay model without departing from the scope of this disclosure.

FIG. 17 illustrates a 3-D model of a lighting fixture with an integratedcamera 1706 according to an example embodiment. In some exampleembodiments, a menu 1704 of 3-D models of lighting fixtures with andwithout integrated IoT devices (e.g., the camera 1706) and standaloneIoT devices may be displayed or otherwise provided on the viewport 106of the AR device 100 and on the corresponding display screen of the ARdevices 120 and 130. Each 3-D model may be associated with a parameterdata file that includes data indicating the field of view 1708 of thecamera 1706. A user may select a 3-D model such as the 3-D model 1702 ofa lighting fixture with the integrated camera 1706 and place the 3-Dmodel on a desired location on the real-time image of a target area asdescribed above with respect to FIGS. 3-11 . The AR devices 100, 120,130 may execute the AR application to overlay on the real-time image ofthe target area a display model (e.g., a polygon or another displaymodel) corresponding to the field of view 1708 of the camera 1706 in asimilar manner as described above with respect to lighting design. Forexample, the AR device 100, 120, 130 may generate the display model orretrieve an existing display model associated with the selected 3-Dmodel 1702. In some example embodiments, the parameter data file mayinclude other information that can be used to generate the display modelwithout departing from the scope of this disclosure.

FIG. 18 illustrates a 3-D model of a lighting fixture with an integratedspeaker or array of speakers 1806 according to an example embodiment. Insome example embodiments, a menu 1804 of 3-D models of lighting fixtureswith and without integrated IoT devices (e.g., the speaker or array ofspeakers 1806) and standalone IoT devices may be displayed or otherwiseprovided on the viewport 106 of the AR device 100 and on thecorresponding display screen of the AR devices 120 and 130. Each 3-Dmodel may be associated with a parameter data file that includes dataindicating the range and/or directionality 1808 of the sound that can beproduced by the speaker or array of speaker or array of speakers 1806,for example, at a maximum rating of the speaker or array of speakers1806 and/or at different percentages of the maximum rating of thespeaker or array of speakers 1806. A user may select a 3-D model such asthe 3-D model 1802 of a lighting fixture with the integrated speaker orarray of speakers 1806 and place the 3-D model on a desired location onthe real-time image of a target area as described above with respect toFIGS. 3-11 . The AR devices 100, 120, 130 may execute the AR applicationto overlay on the real-time image of the target area a display model(e.g., a polygon or another display model) corresponding to the range1808 of the speaker or array of speakers 1806 in a similar manner asdescribed above with respect to the photometric data in lighting design.For example, the AR device 100, 120, 130 may generate the display modelor retrieve an existing display model associated with the selected 3-Dmodel 1802. In some example embodiments, the parameter data file mayinclude other information that can be used to generate the display modelwithout departing from the scope of this disclosure.

FIG. 19 illustrates a 3-D model of a lighting fixture with an integratedmicrophone or array of microphones 1906 according to an exampleembodiment. In some example embodiments, a menu 1904 of 3-D models oflighting fixtures with and without integrated IoT devices (e.g., themicrophone or array of microphones 1906) and standalone IoT devices maybe displayed or otherwise provided on the viewport 106 of the AR device100 and on the corresponding display screen of the AR devices 120 and130. Each 3-D model may be associated with a parameter data file thatincludes data indicating the range and/or directionality 1908 that asound generated at a particular decibel or different decibels can bedetected by the microphone or array of microphones 1906. A user mayselect a 3-D model such as the 3-D model 1902 of a lighting fixture withthe integrated microphone or array of microphones 1906 and place the 3-Dmodel on a desired location on the real-time image of a target area asdescribed above with respect to FIGS. 3-11 .

The AR devices 100, 120, 130 may execute the AR application to overlayon the real-time image of the target area a display model (e.g., apolygon or another display model) corresponding to the range 1908 of themicrophone or array of microphones 1906 in a similar manner as describedabove with respect to the photometric data in lighting design. Forexample, the AR device 100, 120, 130 may generate the display model orretrieve an existing display model associated with the selected 3-Dmodel 1702. In some example embodiments, the parameter data file mayinclude other information that can be used to generate the display modelwithout departing from the scope of this disclosure.

In some example embodiments, the AR devices 100, 120, 130 and the ARapplication may be used to perform lighting as well as IoT design, where3-D models of lighting fixtures with and without IoT devices arepresented to the user on the display screen of the AR devices. Ingeneral, operations provided herein with respect to one of the ARdevices 100, 120, 130 are applicable to other ones of the AR devices100, 120, 130.

In some alternative embodiments, a parameter data file that includesalternative gradient of lighting information may be used instead of thephotometric data file described above. The description provided hereinwith respect to photometric data and photometric data files may beequally applicable to parameter data and parameter data files withalternative gradient of lighting data.

FIGS. 20 and 21 illustrate use of the augmented reality device 100 ofFIG. 1A to simulate sensor-controlled lighting behavior according to anexample embodiment. In some example embodiments, the AR devices 120, 130of FIGS. 1C and 1D may be used instead of the AR device 100. Referringto FIGS. 1A-21 , in some example embodiments, the AR device 100 maydisplay a real-time image 2006 of a target area, for example, a parkinglot or garage or an indoor space in a similar manner as described abovewith respect to FIGS. 3 and 9 . The AR device 100 may display one ormore 3-D models such as a 3-D model 2002 of a lighting fixture with oneor more IoT devices shown as an IoT device 2004. For example, the 3-Dmodel 2002 may correspond to the 3-D model 1602 shown in FIG. 16 or 1702shown in FIG. 16 .

In some example embodiments, the IoT device 2004 may have an operationalrange 2008. For example, the IoT device 2004 may be a sensor such as amotion sensor. To illustrate, the operational range 2008 of the IoTdevice 2004 may be the detection range, angle, or field of view of amotion sensor. As another example, the IoT device 2004 may be a camera,where the operational range 2008 of the IoT device 2004 may be the fieldof view of the camera.

In some example embodiments, some operations of the lighting fixturerepresented by the 3-D model 2002 may depend on or may be controlled bythe one or more IoT devices of the lighting fixtures. To illustrate,after the one or more 3-D models, including the 3-D model 2002 thatincludes the IoT device 2004, are displayed on the viewport 1006, a usercarrying the AR device may move toward the real-time image 2006 and theIoT device 2004 (i.e., toward the 3-D model 2002). When the user reachesthe operational range 2008 (which may or may not be displayed in theviewport 106) of the IoT device 2004, a lighting pattern 2010 may bedisplayed by the AR device 100. The display of the lighting pattern 2010in response to the user moving into or within the operational range 2008of the IoT device 2004 simulates the behavior of the lighting fixturewith one or more IoT devices represented by the 3-D model 2002 inresponse to a person (or a car or other object detected by the IoTdevice) moving into or within the detection or sensing region of the oneor more IoT devices.

In some example embodiments, the lighting pattern 2010 may be removedfrom the viewport 106 in response to the user holding the AR device 100moving out of the operational range 2008 of the IoT device. For example,if the user returns to the original location in the target physicalarea, the image displayed on the viewport 106 may be similar to theimage shown in FIG. 20 .

By simulating the behavior of lighting fixtures without installing thelighting fixtures and the IoT devices, a user may achieve desirableresults, confirm desired operation with the need for physicalinstallation, and/or avoid some design errors. For example, moreaccurate location and/or orientation of IoT devices integrated withlighting fixtures or external to lighting fixtures may be determined bysimulating the behavior of lighting fixtures in response to the IoTdevices.

In some alternative embodiments, the IoT device 2004 may be external tothe lighting fixture represented by the 3-D model 2002. In some exampleembodiments, the behavior of multiple lighting fixtures in response toone or more IoT devices may be simulated in a similar manner. In someexample embodiments, the lighting pattern 2010 may be similar to thelighting pattern shown in FIG. 3 or FIG. 9 . In some exampleembodiments, similar simulation of the operation of devices (that arenot light fixtures, such as automatic doors, shades, fans, thermostats,displays or other controllable devices) being controlled or incommunication with IoT device(s) in response to the AR device enteringor leaving the simulated range or pattern associated with an operatingcharacteristic of an IoT device(s) may be displayed on the AR device.

FIG. 22 illustrates a method 2200 of augmented reality-based lightingand IoT design according to an example embodiment. Referring to FIGS.1A-22 , in some example embodiments, the method 2200 includes, at step2202, displaying, by an augmented reality device (e.g., the AR device100, 120, 130), a real-time image of a target physical area on a displayscreen of the AR device. For example, the AR device 100 may display thereal-time image 304 of the target physical area 302 as viewed by thecamera 102.

At step 2204, the method 2200 may include displaying, by the augmentedreality device, a lighting fixture 3-D model on the display screen inresponse to a user input, where the lighting fixture 3-D model isoverlaid on the real-time image of the target physical area. Forexample, the 3-D model 602 and other 3-D models may be overlaid on thereal-time image 304. To illustrate, the lighting fixture 3-D model maybe overlaid on the real-time image 304 before or after a darkeningfilter has been applied to the real-time image 304 as described withrespect to FIG. 4 . As another example, one or more lighting fixture 3-Dmodels may be overlaid on the real-time image 902 shown in FIG. 9 .

At step 2206, the method 2200 may include displaying, by the augmentedreality device, a lighting pattern on the display screen overlaid on thereal-time image of the target physical area, where the lighting patternis generated based on at least photometric data associated with thelighting fixture 3-D model. For example, image 702, including thelighting pattern, shown in FIG. 7A may be displayed by the AR device 100by overlaying the lighting pattern on the real-time image 304 of thetarget physical area 302. The AR device 100, 120, 130 or another devicemay generate the lighting pattern from at least the photometric dataassociated with the lighting fixture 3-D model as described above.

In some example embodiments, the method 2200 may include darkening thedisplay screen before displaying the lighting fixture 3-D model on thedisplay screen as described with respect to FIGS. 4 and 10 . Forexample, the real-time image of the target physical area may remainvisible after the darkening of the display screen to allow the placementof lighting fixture 3-D models at desired locations in the real-timeimage 304. In some alternative embodiments, the darkening the displayscreen may be omitted when, for example, assessment of lighting patternis not performed. For example, a lighting design may be focused on theaesthetic features of the lighting fixture(s) (or one or more of thelight fixtures subcomponents such as a trim, optic, or accessories) inthe target area instead of lighting patterns.

In some example embodiments, the method 2200 may include changing acolor temperature associated with the lighting pattern displayed on thedisplay screen. The color temperature may be changed in response to auser input. For example, the lighting characteristic selector 1202 maybe used to change and/or select a color temperature as described withrespect to FIGS. 12-13C. In some alternative embodiments, replacing thedisplayed 3-D model by another 3-D model may result in a different colortemperature.

In some example embodiments, the method 2200 may include changing aluminance level associated with the lighting pattern displayed on thedisplay screen. The luminance level may be changed in response to a userinput. For example, the lighting characteristic selector 1202 may beused to change and/or select a luminance level as described with respectto FIGS. 12-13C. In some alternative embodiments, replacing thedisplayed 3-D model by another 3-D model may result in a differentluminance level.

In some example embodiments, the method 2200 may include displaying, bythe augmented reality device, luminance level values indicatingluminance levels associated with the lighting pattern overlaid on thereal-time image of the target physical area, for example, as describedwith respect to FIG. 7B. The method 220 may also include displaying, bythe augmented reality device, one or more other lighting fixture 3-Dmodels on the display screen in response to a user input, for example,as described with respect to FIG. 6 . The one or more lighting fixture3-D models may be overlaid on the real-time image of the target physicalarea in a similar manner as the 3-D model at step 2204. In somealternative embodiments, one or more other lighting fixture 3-D modelsmay be added to the real-time image (e.g., the real-time image 304, 902)displayed on the display screen (e.g., the display screen 106) before orafter darkening the display screen. Alternatively, the darkening of thedisplay screen may be omitted.

In some alternative embodiments, one or more steps of the method 2200may be omitted or may be performed in a different order than describedabove. Although some of the steps of the method 2200 are described withrespect to one or more images or figures, the steps may be applicable toother images and figures without departing from the scope of thisdisclosure. Although some of the steps of the method 2200 are describedwith respect to the AR device 100, the steps may be performed by theother AR devices including the AR device 120 and 130 without departingfrom the scope of this disclosure. In general, the steps of the method2200 may be performed by the AR devices 100, 120, 130. For example, acontroller (e.g., the controller 202) of the AR devices may executesoftware code to perform the steps of the method 2200.

FIG. 23 illustrates a method 2300 of augmented reality-based lightingand IoT design according to another example embodiment. Referring toFIGS. 1A-23 , in some example embodiments, the method 2300 includes, atstep 2302, displaying, by an augmented reality device (e.g., the ARdevice 100, 120, 130), a real-time image of a target physical area on adisplay screen of the AR device. For example, the AR device 100 maydisplay the real-time image 304 of the target physical area 302 asviewed by the camera 102.

At step 2304, the method 2300 may include displaying, by the augmentedreality device, a 3-D model of a lighting fixture with one or more IoTdevices on the display screen in response to a user input, where the 3-Dmodel is overlaid on the real-time image of the target physical area.For example, the 3-D model 602 may correspond to a lighting fixture withone or more integrated IoT devices (or, alternatively, one or morestandalone IoT devices), and the 3-D model 602 and other similar 3-Dmodels may be overlaid on the real-time image 304 shown in FIG. 3 . The3-D model 602 may be overlaid on the real-time image 304 before or aftera darkening filter has been applied to the real-time image 304 asdescribed with respect to FIG. 4 . As another example, a 3-D model of alighting fixture with one or more IoT devices may be overlaid on thereal-time image 902 shown in FIG. 9 .

At step 2306, the method 2300 may include displaying on the displayscreen, by the augmented reality device, a pattern overlaid on thereal-time image of the target physical area, where the patterncorresponds to parameter data associated with the 3-D model. Forexample, the pattern may correspond to the one or more operatingcharacteristics associated with an IoT device(s) integrated with thelighting fixture correspond to the 3-D model. In some exampleembodiments, a lighting pattern as described above, for example, withrespect to FIGS. 7A and 11 may also be overlaid on the real-time imagedisplayed on the display screen.

To illustrate with an example, the one or more IoT devices may includeone or more sensors, and the pattern overlaid on the real-time image mayshow the detection range, angle, and/or field of view of the one or moresensors. For example, a pattern showing the detection range/angle 1608shown in FIG. 16 may be overlaid on the real-time image 304 or 902 in asimilar manner as the lighting pattern or luminance levels describedabove. The 3-D model of the lighting fixture with the one or more IoTdevices may be associated with a parameter data file that includes dataindicating the detection range, angle, and/or field of view of the oneor more sensors.

As another example, the one or more IoT devices may include one or morecameras, and the pattern overlaid on the real-time image may show thefield of view of the one or more cameras. For example, a pattern showingthe field of view 1708 of the camera 1706 shown in FIG. 17 may beoverlaid on the real-time image 304 or 902 in a similar manner as thelighting pattern or luminance levels described above. The 3-D model ofthe lighting fixture with the one or more IoT devices may be associatedwith a parameter data file that includes data indicating the field ofview of the one or more cameras.

As another example, the one or more IoT devices may include one or morespeakers, and the pattern overlaid on the real-time image may show therange and/or directionality of a sound produced by the one or morespeakers, for example, at a particular decibel (a decibel value orvalues may also be displayed). For example, a pattern showing the rangeand/or directionality 1808 of the speaker 1806 shown in FIG. 18 may beoverlaid on the real-time image 304 or 902 in a similar manner as thelighting pattern or luminance levels described above. The 3-D model ofthe lighting fixture with the one or more IoT devices may be associatedwith a parameter data file that includes data indicating the rangeand/or directionality of a sound produced by the one or more speakers atone or more decibels.

As another example, the one or more IoT devices may include one or moremicrophones, and the pattern overlaid on the real-time image may showthe sound detection range and/or directionality of the one or moremicrophones, for example, at a particular decibel (a decibel value orvalues may also be displayed). For example, a pattern showing sounddetection range and directionality 1908 of the microphone 1906 shown inFIG. 19 may be overlaid on the real-time image 304 or 902 in a similarmanner as the lighting pattern or luminance levels described above. The3-D model of the lighting fixture with the one or more IoT devices maybe associated with a parameter data file that includes data indicatingsound detection range and/or directionality of the one or moremicrophones.

In some example embodiments, a lighting pattern as described above, forexample, with respect to FIGS. 7A and 11 may also be overlaid on thereal-time image displayed on the display screen.

In some example embodiments, one or more steps of the method 2300 may beperformed using 3-D models of standalone IoT devices. In some exampleembodiments, one or more steps of the method 2300 may be performed asone or more steps of the method 2200 without departing from the scope ofthis disclosure. In some alternative embodiments, one or more steps ofthe method 2300 may be omitted or may be performed in a different orderthan described above. Although some of the steps of the method 2300 aredescribed with respect to one or more images or figures, the steps maybe applicable to other images and figures without departing from thescope of this disclosure. In general, the steps of the method 2300 maybe performed by the AR devices 100, 120, 130. For example, a controller(e.g., the controller 202) of the AR devices may execute software codeto perform the steps of the method 2300.

FIG. 24 illustrates a method 2400 of augmented reality-based lightingand IoT design according to another example embodiment. Referring toFIGS. 1A-24 , in some example embodiments, the method 2400 includes, atstep 2402, displaying, by an augmented reality device (e.g., the ARdevice 100, 120, 130), a real-time image of a target physical area on adisplay screen of the AR device. For example, the AR device 100 maydisplay the real-time image 304 of the target physical area 302 asviewed by the camera 102. As another example, the AR device 100 maydisplay the real-time image 902 of a target physical area as viewed bythe camera 102.

At step 2404, the method 2400 may include displaying, by the augmentedreality device, a lighting fixture 3-D model on the display screen inresponse to a user input, where the lighting fixture 3-D model isoverlaid on the real-time image of the target physical area. Forexample, the 3-D model 602 may correspond to a lighting fixture with orwithout one or more integrated IoT devices, and the 3-D model 602 andother similar 3-D models may be overlaid on the real-time image 304shown in FIG. 3 . As another example, a 3-D model of a lighting fixturewith or without one or more IoT devices may be overlaid on the real-timeimage 902 shown in FIG. 9 .

At step 2406, the method 2400 may include generating, by the augmentedreality device, a BOM (or purchase order) that includes a lightingfixture corresponding to the lighting fixture 3-D model. For example,the AR device 100 may generate the BOM 808 shown in FIG. 8C. Toillustrate, the AR device 100 may generate or retrieve from a remotedevice (e.g., cloud server) that generates the BOM 808 in response to auser input provided to the AR device 100, for example, via the BOMgeneration input interface 806 or another input interface.Alternatively, the BOM may be generated upon a completion of lightingand IoT design process that may be indicated in one of several means ascan be understood by those of ordinary skill in the art with the benefitof this disclosure.

For example, the 3-D model 602 and other 3-D models may be overlaid onthe real-time image 304. To illustrate, the lighting fixture 3-D modelmay be overlaid on the real-time image 304 before or after a darkeningfilter has been applied to the real-time image 304 as described withrespect to FIG. 4 . As another example, one or more lighting fixture 3-Dmodels may be overlaid on the real-time image 902 shown in FIG. 9 .

In some example embodiments, the method 2400 may include displaying, bythe augmented reality device, a lighting pattern on the display screenoverlaid on the real-time image of the target physical area, forexample, as described with respect to the method 2200. In some exampleembodiments, the method 2400 may include displaying, by the augmentedreality device, a product menu (e.g., the product menu 804 and/or asearch bar to search for products) on the display screen (e.g., theviewport 106) for use by a user to add one or more products to the BOM,such as the BOM 808.

In some example embodiments, the method 2400 may include displaying, bythe augmented reality device, a message (e.g., the design information814) suggesting one or more other lighting products to be added to theBOM (e.g., the BOM 808). In some example embodiments, the method 2400may include determining, by the augmented reality device or viacommunication with a cloud sever, whether one or more products in theBOM (e.g., the BOM 808) are available for purchase or an estimate ofwhen the one or more products may be available for purchase or delivery.In some example embodiments, the method 2400 may include determining, bythe augmented reality device or via communication with a cloud sever,whether one or more products in the BOM (e.g., the BOM 808) arecompliant with an electrical or lighting code or guideline (e.g., ICC,OSHA, Title 24 of the California Code of Regulations, and/or other codesor standards). In some example embodiments, the method 2400 may includedisplaying, by the augmented reality device, a message e.g., the designinformation 814) indicating whether the one or more products in the BOMare compliant with the electrical or lighting code or guideline. Thedisplayed information (e.g., the design information 814) may alsoinclude another message displayed by the AR device suggesting one ormore other lighting products as replacements to one or more productsincluded in the BOM. In some example embodiments, the method 2400 mayalso include displaying a message indicating whether one or morelighting fixtures listed in the BOM provide a light having a lightinglevel that is compliant with an electrical or lighting code orguideline. For example, the message may be included in the designinformation 814 displayed on the viewport 106.

In some example embodiments, one or more steps of the method 2400 may beperformed as one or more steps of the methods 2200 and 2300 withoutdeparting from the scope of this disclosure. In some alternativeembodiments, one or more steps of the method 2400 may be omitted or maybe performed in a different order than described above. Although some ofthe steps of the method 2400 are described with respect to one or moreimages or figures, the steps may be applicable to other images andfigures without departing from the scope of this disclosure. In general,the steps of the method 2400 may be performed by the AR devices 100,120, 130. For example, a controller (e.g., the controller 202) of the ARdevices may execute software code to perform the steps of the method2400.

FIG. 25 illustrates a method of augmented reality-based lighting and IoTdesign according to another example embodiment. Referring to FIGS. 1A-25, in some example embodiments, the method 2500 includes, at step 2502,displaying, by an augmented reality device (e.g., the AR device 100,120, 130), a real-time image of a target physical area on a displayscreen of the AR device. For example, the AR device 100 may display thereal-time image 304 of the target physical area 302 as viewed by thecamera 102. As another example, the

AR device 100 may display the real-time image 902 of a target physicalarea as viewed by the camera 102.

At step 2504, the method 2500 may include displaying, by the augmentedreality device, a 3-D model of a lighting fixture with one or more IoTdevices on the display screen in response to a user input, where the 3-Dmodel is overlaid on the real-time image of the target physical area.For example, the 3-D model 602 may correspond to a lighting fixture (orother device controlled or in communication with one or more IoTdevices) with or without one or more integrated IoT devices, and the 3-Dmodel 602 and other similar 3-D models may be overlaid on the real-timeimage 304 shown in FIG. 3 . As another example, a 3-D model of alighting fixture with or without one or more IoT devices may be overlaidon the real-time image 902 shown in FIG. 9 .

At step 2506, the method 2500 may include displaying on the displayscreen, by the augmented reality device, a lighting pattern overlaid onthe real-time image of the target physical area in response to theaugmented reality device moving within an operational range of the oneor more IoT devices. For example, the lighting pattern may be similar tothe lighting pattern shown in FIGS. 7A and 11 or may be intended to justshow that the 3-D model is emitting a light similar to the emitted lightshown in FIG. 21 . Alternatively, the lighting pattern may have adifferent appearance than shown in FIGS. 7A, 11, and 21 withoutdeparting from the scope of this disclosure. In some exampleembodiments, the method 2500 may include, before displaying the lightpattern, displaying on the display screen an IoT device pattern overlaidon the real-time image of the target physical area in response to theaugmented reality device moving within an operational range of the oneor more IoT devices. For example, the IoT device pattern may correspondto the operational range 2008 (e.g., range, angles, field of view, etc.)of the one or more IoT devices.

In some example embodiments, the method 2500 may include displaying onthe display screen, by the augmented reality device, an IoT devicepattern overlaid on the real-time image of the target physical area, forexample, as shown in FIG. 21 . For example, the one or more IoT devicesmay include one or more motion sensors, where the operational range ofthe one or more IoT devices is a detection range of the one or moremotion sensors such as the detection range/angles/field of view 1608 ofthe sensor 1606 shown in FIG. 16 or the operational range 2008 (e.g.,detection range) of the one or more IoT devices 2004 (e.g., one or moresensors) shown in FIG. 20 . As another example, the one or more IoTdevices may include one or more cameras, where the operational range ofthe one or more IoT devices is a field of view of the one or morecameras such as the field of view 1708 of the camera 1706 shown in FIG.17 or the operational range 2008 (e.g., field of view) of the one ormore IoT devices 2004 (e.g., one or more cameras) shown in FIG. 20 .

In some example embodiments, the method 2500 includes removing theoverlaid lighting pattern from the display screen in response to the ARdevice moving out of the operational range of the one or more IoTdevices. For example, when a person carrying the AR device 100 movesoutside of the operational range 2008 of the one or more IoT devices(e.g., one or more motion sensors and/or cameras), the light patternillustrated in FIG. 21 may be removed such that, when the AR device 100is returned to the same position as in FIG. 20 , the image displayed onthe AR device 100 appears similar to the image shown in FIG. 20 .

In some example embodiments, one or more steps of the method 2500 may beperformed to simulate the operation of devices (that are not lightfixtures, such as automatic doors, shades, fans, thermostats, displaysor other controllable devices) being controlled or in communication withIoT device(s) in response to the AR device entering or leaving thesimulated range or pattern associated with an operating characteristicof an IoT device. In some example embodiments, one or more steps of themethod 2500 may be performed as one or more steps of the other methodsdescribed herein without departing from the scope of this disclosure. Insome alternative embodiments, one or more steps of the method 2500 maybe omitted or may be performed in a different order than describedabove. Although some of the steps of the method 2500 are described withrespect to one or more images or figures, the steps may be applicable toother images and figures without departing from the scope of thisdisclosure. In general, the steps of the method 2500 may be performed bythe AR devices 100, 120, 130. For example, a controller (e.g., thecontroller 202) of the AR devices may execute software code to performthe steps of the method 2500.

In the above description, a display model that represents photometricdata or other parameter data associated with one or more lightingfixtures or parameter data associated with one or more IoT devices maybe a 2D polygon, a 3-D polygon, a combination of 2D and/or 3-D polygons,graphical image(s), another type of image(s), etc. A polygon as anexample of a display model may be a 2D polygon, a 3-D polygon, acombination of 2D and/or 3-D polygons, graphical image(s), another typeof image(s), etc.

In some alternative embodiments, another device may perform some of theoperations described herein with respect to the AR device 100. Toillustrate, another device, such as a local or remote server, maygenerate one or more display models based on information provided by theAR device 100. For example, the AR device 100 may provide information,such as information indicating selected lighting fixture 3-D model,and/or relevant photometric data or other parameter data to anotherprocessing device that generates the display model(s), and the AR device100 may receive/retrieve the generated display model(s) from the otherprocessing device.

Although particular embodiments have been described herein in detail,the descriptions are by way of example. The features of the exampleembodiments described herein are representative and, in alternativeembodiments, certain features, elements, and/or steps may be added oromitted. Additionally, modifications to aspects of the exampleembodiments described herein may be made by those skilled in the artwithout departing from the spirit and scope of the following claims, thescope of which are to be accorded the broadest interpretation so as toencompass modifications and equivalent structures.

What is claimed is:
 1. An augmented reality-based lighting designmethod, comprising: detecting, by an augmented reality device, anexisting light fixture installed in an area displayed on a displayscreen; determining, by the augmented reality device, one or moresurfaces in the area using image data; determining, by the augmentedreality device, one or more replacement light fixtures at least based ona type of the existing light fixture and the one or more surfaces; anddisplaying, on a display screen of the augmented reality device, alighting fixture model of a replacement light fixture from among the oneor more replacement light fixtures, and wherein the lighting fixturemodel is overlaid on a real-time image of the area displayed on thedisplay screen.
 2. The method of claim 1, further comprising displayingilluminance values overlaid on the one or more surfaces of the area inthe real-time image of the area.
 3. The method of claim 2, wherein oneor more of the illuminance values are determined based on at leastphotometric data associated with the lighting fixture model of thereplacement light fixture and spatial relationship between the one ormore surfaces and the replacement light fixture model.
 4. The method ofclaim 1, wherein the lighting fixture model of the replacement lightfixture is overlaid on the real-time image at a location of the existinglight fixture in response to a user input provided to the augmentedreality device selecting the lighting fixture model of the replacementlight fixture from among one or more lighting fixture models of the oneor more replacement light fixtures.
 5. The method of claim 1, whereindetermining the one or more replacement light fixtures is performedfurther based on a spatial relationship between the existing lightfixture and the one or more surfaces.
 6. The method of claim 1, furthercomprising: determining, by the augmented reality device, a second oneor more replacement light fixtures at least based on a type of a secondexisting light fixture and the one or more surfaces; and displaying,overlaid on the real-time image of the area displayed on the displayscreen, a second lighting fixture model of a second replacement lightfixture from among the second one or more replacement light fixtures,wherein the second existing light fixture is replaceable by the secondreplacement light fixture.
 7. The method of claim 6, further comprisingdisplaying illuminance values overlaid on the one or more surfaces ofthe area displayed on the display screen, wherein the illuminance valuesare determined based on at least photometric data associated with thelighting fixture model, photometric data associated with the secondlighting fixture model, and spatial relationships of the light fixturemodel and the second light fixture model with the one or more surfaces.8. The method of claim 1, further comprising, determining, by theaugmented reality device, the type of the existing light fixture basedon at least the image data, wherein a type of the replacement lightfixture is same as the type of the existing light fixture.
 9. The methodof claim 1, further comprising, determining, by the augmented realitydevice, the type of the existing light fixture based on at least theimage data, wherein a type of the replacement light fixture is differentthan the type of the existing light fixture.
 10. The method of claim 1,further comprising, measuring, by a sensor included on the AR device, anilluminance value observed in the area.
 11. The method of claim 1,further comprising, generating a bill of material that includes anidentifier of the replacement light fixture.
 12. An augmented reality(AR) device, comprising: a camera to capture a real-time image of anarea; a sensor; a display screen; a memory device storing a softwarecode; and a controller configured to execute the software code to:detect an existing light fixture installed in the area using image datafrom the camera; determine one or more surfaces in the area using thesensor and the image data from the camera; determine one or morereplacement light fixtures at least based on a type of the existinglight fixture and the one or more surfaces; and display, overlaid on thereal-time image of the area displayed on the display screen, a lightingfixture model of a replacement light fixture from among the one or morereplacement light fixtures.
 13. The AR device of claim 12, wherein thecontroller is further configured to display illuminance values overlaidon the one or more surfaces displayed on the display screen.
 14. The ARdevice of claim 13, wherein the controller is further configured todetermine the one or more of the illuminance values based on at leastphotometric data associated with the replacement light fixture andspatial relationship between the replacement light fixture and the oneor more surfaces.
 15. The AR device of claim 12, wherein the controlleris further configured to determine the type of the existing lightfixture based on at least the image data from the camera.
 16. Anaugmented reality-based lighting design method, comprising: determining,by the augmented reality device, one or more surfaces in the area usingimage data; determining, by the augmented reality device, one or morereplacement light fixtures at least based on the one or more surfaces;and displaying, on a display screen of the augmented reality device, alighting fixture model of a replacement light fixture from among the oneor more replacement light fixtures, and wherein the lighting fixturemodel is overlaid on a real-time image of the area displayed on thedisplay screen.
 17. The method of claim 16, further comprisingdisplaying illuminance values overlaid on the one or more surfaces ofthe area in the real-time image of the area.
 18. The method of claim 17,wherein one or more of the illuminance values are determined based on atleast photometric data associated with the lighting fixture model of thereplacement light fixture and spatial relationship between the one ormore surfaces and the replacement light fixture model.
 19. The method ofclaim 17, further comprising, measuring, by a sensor included on the ARdevice, an illuminance value observed in the area.
 20. The method ofclaim 16, further comprising, generating a bill of material thatincludes an identifier of the replacement light fixture.